Automatic Feature Induction for Text Classification

The Problem: All classifiers require a set of features that can be used to distinguish between different examples. In some cases, such as determining whether a chess position is a winning position, the features are clear (the positions of the chess pieces). In other cases, such as text, they are less clear. A document is simply a string of characters. Standard practice dictates that documents should be broken into words, common, uninformative words should be removed and a stemmer is to be applied for removal of common endings (such as “-s” and “-ing”). However, this processing is insufficient for many languages and many modern-day classification problems. For example, documents about the XWindow system include an unusually large number of words that begin with the letter “x,” but traditional processing obscures this fact. Unsolicitied commercial e-mail (“spam”) can often be identified by the use of foreign languages, a disproportionate number of exclamation points or capital letters. Again, such information will often be discarded.